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REMARKS 

This Preliminary Amendment cancels original claims 1 to 10, and adds new claims 
1 1 to 20 in the underlying PCT Application No. PCT7DE03/00452. The new claims conform the 
claims to the U.S. Patent and Trademark Office rules and do not add new matter to the application. 

In accordance with 37 C.F.R. § 1.125, the Substitute Specification (including the 
Abstract, but without the claims) contains no new matter. The amendments reflected in the 
Substitute Specification (including the Title and Abstract) are to conform the Specification and 
Abstract to U.S. Patent and Trademark Office rules or to correct informalities. As required by 37 
C.F.R. § 1.121 and § 1.125, a Marked Up Version Of The Substitute Specification comparing the 
Specification of record and the Substitute Specification also accompanies this Preliminary 
Amendment. In the Marked Up Version, underlining indicates added text and "strike-throughs" 
and double-brackets indicate deleted text. Approval and entry of the Substitute Specification 
(including Abstract) is respectfully requested. 

The underlying PCT Application No. PCT/DE03/00452 includes an International 
Search Report, dated June 30, 2003, a copy of which is included. The Search Report includes a 
list of documents that were considered by the Examiner in the underlying PCT application. 

It is asserted that the subject matter of the present application is new, non-obvious, 
and useful. Prompt consideration and allowance of the application are respectfully requested. 
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[10191/3651] 
METHOD FOR DETECTING A PERSON IN A SPACE 

Background Information FIELD OF THE INVENTION 

The present invention is directed to a method for detecting a 
person in a space according to the species defined in the 
independent claim . ^ 

Advantages of the Invention BACKGROUND INFORMATION 
The method according to the present invention for detecting a 
person in a space having the features of the independent claim 
has may have the advantage of being a compromise between 
complexity and simplicity: Complex situations, such as persons 
in unusual poses, with unusual clothes, or persons that are 
partially concealed, are processed with little effort. In this 
connection, it should be noted that the "detection of a 
person" is understood to also include the detection of 
specific body parts. 

During the search, the presented method concentrates on 
comparatively few body parts that can be detected in a 
reliable and fast manner, such as the head. If the head of an 
occupant is in an unusual position in the motor vehicle, or if 
there are two occupants per seat in the motor vehicle, or if 
there exist partial concealments of the body, the respective 
heads are still found because there is no need to bring a 
complete human model into line with the scene. 

The method provides different sub-models for a body part, the 
different sub-models being a function of the method in the 
case of different clothing and headgear. 

Therefore, the method of the present invention is superior 
both to the model -free method and to a complex, complete CAD 
model because it can also cope with atypical scenarios. 
"Model -free" means that here the volume distribution, or other 
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characteristics such as the curvature or texture, are directly 
evaluated by a classifier. A serious disadvantage of this 
method is that not all situations (such as the pose or 
clothing) can be covered in the training. This results in poor 
5 detection performance for model -free approaches. 

Unlike the CAD model, the method according to the present 
invention is fast because only a few parameters are estimated. 
For a fast algorithm, -art — is also advantageous that not rather 

10 than all measured values, [[but]] it may be that only those of 
the volume to be examined may have to be taken into account to 
adapt a sub-model . The method according to the present 
invention has the characteristic that a hierarchical model of 
a human is adapted to the spatial data of a depth sensor. This 

15 depth sensor preferably , for example, has high resolution, -frr 
particular For example , the organization of the model and the 
method of adaptation to the spatial data of the depth sensor 
[[are]] may be advantageous. In this context, the model [[is]] 
may be a shape model (surface model) of the human body. The 

20 model [[is]] may be designed to be articulatable, i.e., 

movable, and to be able to describe different forms of bodies 
and clothing. 

Advantageous improvements of the method for detecting a person 
25 in a space described in the independent c laim are made 

possible by the measures and further refinements s p ecified in 
the dependent claims . 

— is particularly advantageous that According to an example 
30 embodiment , the complete model [[is]] may be adapted to follow 
the person over time by further adapting the sub-models using 
the data at predetermined intervals. This allows a so-called 
"tracking"; a complete adaptation no longer being needed, but 
the detected body parts being trackable in space and time. 

35 

*3rt — is als o advantageous that According to an example 
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embodiment , at least one body part, such as the head, the 
shoulder, or also, for example, the chest of a human, is of 
particular importance for the classification of an occupant, 
especially in automotive applications such as restraint 
5 systems or occupant condition monitoring. Moreover, these body 
parts are particularly may be easy to recognize. 

Moreover, it [ [is] ] may be beneficial that intensity 
information from the data is used, the intensity information 

10 being texture characteristics, such as gray levels of the gray 
image or color values of the color image, which are each 
determined by an imaging sensor. These texture characteristics 
may then be associated with individual body parts in order to 
associate more detailed information with the individual body 

15 parts. This also allows consistency checking to be able to 
detect a person. 

However, the general consistency check [ [is] ] may be performed 
using the position parameters that define the relationship 

2 0 between the individual sub-models associated with the 

different body parts. This consistency check allows the 
adaptation of the sub-models by the measured sensor values to 
lead to reasonable values changes. 

25 Moreover, it is an advantage that the complete model [ [is] ] 

may be transmitted for occupant classification to a restraint 
system in a vehicle in which the person is located. This then 
allows, in particular for example , detection of persons; the 
system primarily detecting the posture of the occupant to 

3 0 identify a dangerous position of the occupant with respect to 

air bags and belt pretensioners . Thus, the so-called 
"out-of -position positions" are identified to prevent 
activation of the air bags in such positions in order to avoid 
injuries. The so-called "tracking" allows dynamic 
35 out-of -position sensing. Thus, the posture of the occupant is 
detected in real time. This [[plays]] may play a role 
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especially in the case of fast movements of the head relative 
to the sensor, which are induced, for example, by vehicle 
decelerations during braking maneuvers or collisions. Here, it 
is also possible to classify the occupants based on size or 
5 biometric conditions, such as the average head radius or the 
shoulder width. 

Another use of the method according to the present invention 
is the anti-pinch protection for extremities or objects in 

10 power windows. Extremity detection can also be used for 

gesture recognition. Monitoring the passenger compartment 
while the vehicle is parked [ [can] ] may also be an application 
of the method according to an example embodiment of the 
present invention. This may be used, in particular for 

15 example , in a car alarm system. It is also possible to 

automatically operate the child security door lock against 
accidental opening if a child has been detected by the method 
according to the present invention. In combination with a 
temperature sensor, it is possible to adjust the temperature 

20 by opening and closing the windows, or by controlling the air 
conditioning, in case a person or, in particular for example , 
a child is alone in the vehicle. Convenience features [[can]] 
may also benefit from an example embodiment of the method 
according to the present invention. The adjustment of 

2 5 headrests, mirrors, sun visors, steering wheel and/or 

contoured seats can be carried out automatically. The control 
of other systems that require the exact position of the head 
is rendered possible, inter alia, a hands -free kit, head-up 
displays, or noise control systems. Other methods, for 
30 example, for identifying the occupants, or eye localization 

are aided by a precise localization of the head, if necessary, 
along with an orientation measurement. 

The depth sensor takes the form, in particular for example , of 

3 5 an image pickup. Above all, a video sensor, such as a stereo 

camera system or a mult i -camera system, may be used here. Also 
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usable here are a distance-measuring matrix sensor using the 
transit-time principle or the phase-measuring principle, a 
laser scanner, a laser radar, or structured lighting. Further, 
the distance-measuring matrix sensor may use a microwave 
5 radar, or the measuring principle may be based on the emission 
of sound waves (ultrasound) , or an arbitrary electromagnetic 
radiation (visible light, invisible light, microwaves). 

Likewise, the representation of the sub-models contained in 
10 the modules can take completely different forms, such as CAD 
data, measurement data, triangulated data, superquadrics , or 
two-dimensional descriptions . 

Drawing 

15 

Exemplary embodiments of the present invention are depicted in 
the drawing and will be explained in greater detail — in the 
following description . — In the drawing, 
BRIEF DESCRIPTION OF THE DRAWINGS 

2 0 Figure 1 shows a block diagram of an example embodiment of a 

device according to the present invention [[;]] _s_ 

Figure 2 is a first flow chart of an example embodiment of the 
method according to the present invention! [;] 

25 

Figure 3 shows a block diagram of an example embodiment of the 
method according to the present invention [[;]] ^_ 

Figure 4 is a representation of an example embodiment of the 

3 0 hierarchical models [[;]] _-_ 

Figure 5 shows a shape model of the head in different 
variations and depths of detail , according to an example 
embodiment of the present invention. - ; — 

35 

Figure 6 is a second flow chart of an example embodiment of 
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the method according to the present invention-; — artd^ 

Figure 7 shows different mounting positions to cover the rear 
passenger compartment , according to an example embodiment of 
5 the present invention. - — 

Description DETAILED DESCRIPTION 

Systems for sensing the passenger compartment of a motor 
vehicle are based on very different physical measuring 

10 principles. For example, it is possible to use a weight mat 

here; the pressure profile being used to determine whether the 
seat is empty or occupied. In further refinements 
Alternatively , it is possible to carry out a classification. 
However, the sensor systems relevant to the method according 

15 to the present invention are depth-measuring sensor systems. 
Here, the distance-measuring matrix sensor preferably , for 
example , takes the form of an image pickup. Such an image 
pickup may be a video sensor, a radar sensor, an ultrasound 
sensor, or a multi-camera system. In this connection, it is 

20 possible to use different measuring principles: the 

transit-time principle, the phase -measuring principle, the 
principle of structured lighting, and the laser scanner 
principle . 

25 The spatial data of the depth sensor, which generally takes 
the form of a matrix sensor and produces more than a hundred 
measured values virtually simultaneously, is used to detect a 
person in a space. In the case of the passenger compartment of 
a motor vehicle this is, in particular for example , the 

3 0 occupant -presence situation. Besides the pure occupant 

presence, it [ [is] ] may also be necessary to classify the 
occupants, i.e., size, pose, and weight. In this connection, 
especially the localization of individual body parts, such as 
the head or the chest, is of [[great]] interest for the 

35 control of, for example, restraint systems. Such information 
is also useful for various convenience features, such as the 
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adjustment of rearview mirrors. 



Figure 1 shows an example embodiment of a device according to 
the present invention in a block diagram. A person 6 is 
5 detected by a depth sensor 1 in terms of his/her spatial 

extent. Depth sensor 1 is mounted, for example, in a passenger 
compartment to act as an image pickup. The spatial data are 
transmitted to a processor 2 on which the method according to 
the present invention is executed. A restraint system 3, an 
10 ant i -pinch function 4, and convenience features 5 are 

controlled by processor 2 as a function of the result of the 
method according to the present invention. Therefore, 
processor 2 may be located in a control unit. 

15 Further vehicle functions are also controllable by processor 2 
so that these vehicle functions may also benefit from the 
inventive method for detecting a person. However, — the The 
method according to the present invention is also of great 
interest applicable outside of motor vehicles, for example, 

20 for video surveillance. Access systems may also benefit — from 
utilize the method according to the present invention. 

Figure 2 shows an example embodiment of the sequence of the 
inventive method in a first flow chart. In method step 6, 

2 5 sensor 1 produces sensor values that provide spatial data 

about person 6. This data is transmitted to processor 2. 
Processor 2 loads on its memory a model of a human which is 
hierarchically structured; i.e., there exist sub-models for 
different body parts, the sub-models in turn being divided 
30 into further sub-models. This subdivision is done to allow 
optimal adaptation for different typical clothing situations 
of the particular body. Then, processor 2 selects the 
appropriate sub-model at the lowest level of the sub-models, 
and performs an adaptation using the spatial data. For this 

3 5 purpose, any optimization method may be employed. 
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Method step 8 checks whether this adaptation is consistent 
with respect to the position parameters. If this not the case, 
then the method returns to step 7 to repeat the adaptation. 
However, if this is the case, then, in method step 9, the 
5 complete model is gradually assembled from the sub-models. If, 
in the process, certain body parts are not found, for example, 
because of a concealment effect, a consistent interpretation 
of the spatial data is still obtained. This complete model is 
then used to recognize the person. This data is then made 
10 available to other functions, such as the restraint system, 
the convenience feature, or the ant i -pinch protection. 

Figure 3 explains shows an example embodiment of the inventive 
method in a block diagram. A person 11 is seated on a vehicle 

15 seat 10, and is spatially detected by a depth sensor 12. This 
depth sensor 12 takes the form of a video sensor here, and is 
located above the windshield. However, depth sensor 12 can 
also be installed at other locations in the passenger 
compartment. The spatial measured values of sensor 12 are 

2 0 transmitted to a processor 13 which adapts the spatial model 
to the spatial measured values using the sub-models. To this 
end, a complete model 14 is loaded from a memory. Processor 13 
delivers the size, position, and posture of the occupant as an 
output quantity 15. 

25 

In Figure 4, the hierarchical model is explained shown by way 
of an example embodiment . The complete model is referred to as 
"shape model 0 16". This model 16 is further divided into one 
or more units 17 through 19, which are referred to as shape 

30 model 1 through shape model 3. The permitted relationships 
between these sub-models, which are defined by the relative 
position parameters, are stored for these sub-models. 
Analogously to the division of shape model 0 into its 
sub-models, it is possible to break down these sub-models into 

35 sub-models as well, and to specify the relationships of the 

sub-models. The sub-models of shape model x are referred to as 
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"shape model x.y" . In this connection, y denotes a non-zero 
natural number. The model may contain an arbitrary number of 
such levels. The number of the further subdivisions and the 
depth of the subdivisions, i.e., the number of levels, are not 
5 predetermined and may be handled differently for different 

sub-models. All branches of the model have in common that they 
contain one or more surface models as the last level. The set 
of surface models describes the respective body part in 
different forms and depths of detail. The shape model 1.17 

10 sub-model is composed of sub-models for head 27. Shape model 
2.18 is divided into shape models 2.1.20 and 2.2.21 of which 
shape model 2.1 pertains to upper arm 28 while shape model 2.2 
pertains to lower arm 29. Shape model 3.19 is divided into 
three shape models 3.1.22, 3.2.23 and 3.3.24, which are 

15 defined by the relative position parameters with respect to 
each other. Shape model 3.1.22 is divided into shape models 
3.1.1 and 3.1.2, which are provided with reference numerals 25 
and 26, respectively. Shape model 3.1.1 is divided into 
further sub-models 30 that define the shoulder here. Shape 

20 model 3.1.2 defines different pieces of torso information. 
Likewise, body parts 32 and 33 are associated with shape 
models 3.2.2 3 and 3.3.24, respectively. The number of the 
further subdivisions and the depth of the subdivisions, i.e., 
the number of levels, are not predetermined and may be handled 

25 differently for different sub-models. All branches of the 
model have in common that they contain one or more surface 
models as the last level. The set of surface models describes 
the respective body part in different forms and depths of 
detail. During the adaptation of the model to the scene, the 

30 model is processed starting from certain shape models or 
individual surface models. These individual parts are 
assembled to form the complete model, taking into account the 
consistency with respect to the position parameters. Once the 
rough position parameters have been determined, detailed 

35 surfaces may be further adapted, taking into account the 
consistency conditions. 
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If certain body parts are not detected, for example, because 
of concealment effects, a consistent interpretation of the 
spatial measured data is still obtained. 



5 An exemplary set of surface models of the head is represented^, 
according to an example embodiment, is shown in Figure 5 . The 
complexity of the model variants increases on abscissa 34. An 
increasing depth of detail of the model is described on 
ordinate 35. The simplest model 36 becomes the more complex 

10 sub-model 37, which is sub-divided into a more detailed model 
38 and an even more detailed model 40; the more detailed model 
40 already representing the head shape fairly accurately. For 
model 38, there exist model 39 which has the same depth of 
detail, but greater complexity. Selecting a greater depth of 

15 detail for model 3 9 leads to model 41, which shows a head with 
a hat that has a brim. Sub-models 4 0 and 41 are ultimately 
used for adaptation to the spatial data. 

Figure 6 [ [is] ] shows a second flow chart of an example 
2 0 embodiment of the method according to the present invention. 

In method step 43, depth sensor 1 produces the spatial data. 
Method step 45 defines a region to be examined. This is done 
using data of already processed shape models: the detection 
status, position, and position parameters from method step 44. 
25 In method steps 46 and 47, the model is adapted to the spatial 
data, taking into account the spatial shape model from method 
step 42. This step may be carried out, for example, using a 
standard method of model adaptation in terms of the minimum 
square error . 

30 

If this adaptation was successful, then, in method step 48, 
the detection status is stored in terms of position and 
position parameters. In method step 49, the position and 
position parameters of the model are then checked for 
35 consistency. To this end, the data of already processed shape 
models from method step 44 is reused. In method step 50, the 
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results of the adaptation are recorded in the detection status 
of the sub - mode 1 . The detection status and the position 
parameters are passed on to the next higher level. If, again, 
relationships exist with another model, then a consistency 
5 check is performed until the complete model has been 

assembled. Once the model has been adapted, a subsequent, 
complete adaptation is no longer needed, and the detected body 
parts can be tracked in space and time. For fast and stable 
implementation, only body parts that can be detected quickly 

10 and reliably should be included in the model, typically the 
head, torso, and arms. The head module and the upper part of 
the body play a central role in the model because the human 
head is of outstanding importance to the set objective and, in 
addition, is comparatively invariant in shape. Moreover, the 

15 head is usually spatially exposed, and can therefore be 

located relatively reliably. The presented method is generally 
independent of the source of the spatial data. 

Figure 7, image (a), shows an additional sensor for covering 
2 0 the rear passenger compartment , according to an example 

embodiment of the present invention . Figure 7b, in contrast, 
shows an alternative mounting position of the sensor for 
covering the entire rear passenger compartment , according to 
an example embodiment of the present invention . This is 
25 denoted by reference numeral 52, whereas in Figure 7a, sensor 
53 is provided for the front seat and sensor 51 for the rear 
seat . 
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Abstract ABSTRACT OF THE DISCLOSURE 

Pro p osed is a A method for detecting a person in a space is 
disclosed, including- ; — at least one depth sensor producing 
spatial data about the space to be monitored^ — at least one 
5 sub-model which is subdividable into further sub-models being 
used for at least one selected body part of a humane — the 
spatial data being used to adapt the sub-models^i — the 
adaptation being checked by position parameters between the 
sub-models of different body parts, and the person being 
10 recognized using a complete model made up of the checked 
sub -mode Is . 

(Figure 3) 
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